@bentsherman Let's aim moving this to "Ready"

I am testing the entire flow with the registry locally. Before we merge this PR, we need to:

1. Finalize and merge https://github.com/seqeralabs/plugin-registry/issues/171
2. Finalize and merge Generate plugin spec nextflow#6361 (I need to review the schema structure)
3. Release Nextflow 25.08.0-edge

You should be able to test everything in the local dev. Check this for using unreleased plugin https://github.com/nextflow-io/nextflow/blob/a5c19b895a5d64ef4ae76907b5f38dc7074df22e/settings.gradle#L17-L19

@pditommaso this is ready to merge, tested with registry-dev and Nextflow 25.09.0-edge

I'm confused. This project doesn't even use Gradle 9:

I think that's why I'm not getting a build error

As far as I can tell there is no circular dependency:

1. build the nextflow plugin
2. run the PluginSpecWriter from nextflow with the plugin JAR and user-defined extension points to build the spec
3. package and publish the plugin

In other words the plugin JAR is built before the spec file, and the same nextflow runtime specified by the plugin is used to build the spec. For this reason it only builds the spec if the nextflow version is >=25.09.0-edge

The problem arise when using from nextflow build

I see. Well, this is a good example where mavenLocal works better 😄

it remains a mess, for example when trying to make 25.10.0 and having a plugin pinning that version, the gradle plugin will fail to resolve it

You mean for core plugins? The core plugins don't need to generate plugin specs, for that case we could just disable it

@pditommaso I added a flag to not build the plugin spec, which should be used by the core plugins. I also fixed the issues with Gradle 9 with some help from Claude. Let me know if it works for you

IMO it would be preferable a solution that works the same both for "internal" and external plugins.

Exploring the annotation processor i've ended up with this solution. Have a look.

It looks like your solution is more or less the same. You still use a class loader with the Nextflow runtime and the plugin JAR, only now you are loading all of the annotation classes by name instead of just the plugin spec entrypoint (nextflow.plugin.spec.PluginSpecWriter)

The problem here is that now the code for generating the JSON metadata is duplicated in Nextflow and the Gradle plugin. You still need this code in the Nextflow runtime to generate this metadata for the language server

So if you want something that works for both core plugins and community plugins, the solution is what we merged in nextflow-io/nextflow#6361 in combination with this PR.

• The code for generating spec definitions is in the Nextflow runtime
• The language server uses it at build time to extract core definitions (core runtime + core plugins)
• The Gradle plugin uses it at plugin build time to extract a plugin spec
  • core plugins should skip this part because they don't need plugin specs

Let me know if you encounter any more issues using this PR

Another option could be moving this project into the nextflow repo. That would allow the plugin to stay in sync core codebase.

I'm not sure how that would work. The Gradle plugin is intended to be used with any version of Nextflow, but moving it into the Nextflow codebase would tie it to a specific Nextflow version

In the codebase it uses the "inline" version. Other plugins the canonical distribution

That's essentially what I have achieved in this PR -- the core plugins should just use useDefaultDependencies = false and buildSpec = false. We could collapse these into a single flag like corePlugin if you prefer.

For reference, the only alternative I ever came up with is the approach currently used by the language server:

https://github.com/nextflow-io/language-server/blob/b7d958a91da474485673495fa7ee8360723bfb44/build.gradle#L70-L85

The language server defines the buildSpec task, which runs a Groovy script (SpecWriter) under src/spec. This SpecWriter script performs the same function for the language server that PluginSpecWriter does for third-party plugins.

This approach is fine for the language server, but for third-party plugins it would break the "pure config" interface that we currently have with the Gradle plugin. Plugin devs would have to maintain a copy of PluginSpecWriter in every plugin repo, back to the old days of plugin boilerplate.

Now I'm getting this

» ./gradlew plugins:nf-amazon:releasePluginToRegistryIfNotExists

> Task :plugins:nf-amazon:compileGroovy
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /Users/pditommaso/Projects/nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :plugins:nf-amazon:buildSpec FAILED
Exception in thread "main" java.lang.NoClassDefFoundError: software/amazon/awssdk/services/s3/model/ObjectCannedACL
        at java.base/java.lang.Class.getDeclaredFields0(Native Method)
        at java.base/java.lang.Class.privateGetDeclaredFields(Class.java:3473)
        at java.base/java.lang.Class.getDeclaredFields(Class.java:2542)
        at nextflow.config.spec.SpecNode$Scope.of(SpecNode.java:200)
        at nextflow.config.spec.SpecNode$Scope.of(SpecNode.java:214)
        at nextflow.plugin.spec.PluginSpec.build(PluginSpec.groovy:56)
        at nextflow.plugin.spec.PluginSpecWriter.main(PluginSpecWriter.groovy:45)
Caused by: java.lang.ClassNotFoundException: software.amazon.awssdk.services.s3.model.ObjectCannedACL
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
        ... 7 more

If I add buildSpec = false, I get another error

» ./gradlew plugins:nf-amazon:releasePluginToRegistryIfNotExists

> Task :plugins:nf-amazon:buildSpec FAILED
Error: Could not find or load main class nextflow.plugin.spec.PluginSpecWriter
Caused by: java.lang.ClassNotFoundException: nextflow.plugin.spec.PluginSpecWriter

The first is the more problematic, I understand for nextflow plugins it should be used turned off, but the same problem can arise for third party plugins

With this fix I am able to build Nextflow even without buildSpec = false

Have you tried it? I'm getting this error

» ./gradlew plugins:nf-azure:releasePluginToRegistryIfNotExists

> Task :plugins:nf-azure:compileGroovy
> Task :plugins:nf-azure:releasePluginToRegistryIfNotExists FAILED

FAILURE: Build failed with an exception.

* What went wrong:
A problem was found with the configuration of task ':plugins:nf-azure:releasePluginToRegistryIfNotExists' (type 'RegistryReleaseIfNotExistsTask').
  - In plugin 'io.nextflow.nextflow-plugin' type 'io.nextflow.gradle.registry.RegistryReleaseIfNotExistsTask' property 'specFile' specifies file '/Users/pditommaso/Projects/nextflow/plugins/nf-azure/build/resources/main/META-INF/spec.json' which doesn't exist.
    

I thought I added a check for that, but guess i forgot. I will fix it

Okay try again

Ben, please try it otherwise this will never end

./gradlew plugins:nf-azure:releasePluginIfNotExists
> Task :plugins:nf-azure:releasePluginToRegistryIfNotExists FAILED

[Incubating] Problems report is available at: file:///Users/pditommaso/Projects/nextflow/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* What went wrong:
A problem was found with the configuration of task ':plugins:nf-azure:releasePluginToRegistryIfNotExists' (type 'RegistryReleaseIfNotExistsTask').
  - In plugin 'io.nextflow.nextflow-plugin' type 'io.nextflow.gradle.registry.RegistryReleaseIfNotExistsTask' property 'specFile' specifies file '/Users/pditommaso/Projects/nextflow/plugins/nf-azure/build/resources/main/META-INF/spec.json' which doesn't exist.
    
    Reason: An input file was expected to be present but it doesn't exist.
    
    Possible solutions:
      1. Make sure the file exists before the task is called.
      2. Make sure that the task which produces the file is declared as an input.
    
    For more information, please refer to https://docs.gradle.org/9.1.0/userguide/validation_problems.html#input_file_does_not_exist in the Gradle documentation.

Okay, I see how you're testing it now. Sorry for the back and forth. With this change I was able to run the release task for nf-azure

It was still failing on my side 👉 fa585b9 🤷‍♂️

Refactored a bit and added more docs as reference